The present disclosure relates generally to event queuing, and more particularly to a scalable event distribution system useful in a variety of computer contexts.
Event distribution is a common problem in network computing environments. Network nodes often need to communicate information to other network nodes. One solution is for each node in the network to be directly connected to every other node, and to send events itself to the other nodes. This approach becomes extremely complex with even a relatively small number of nodes, as the number of connections grows exponentially. The amount of management necessary to account for nodes joining and leaving the network is far from trivial.
Many products and standards attempt to solve this problem by providing an event service. Typical event services include a centralized manager (or a management protocol) and one or more named channels. Communications happen asynchronously by putting events into the channel; threads or applications that want to receive events “listen” for events and receive them from the channel.
Typical event services are of one of two types: a queue or a pipe. A queue-based event service holds information in storage until it is read by one of a number of clients. When a client confirms that the information has been appropriately read or processed, the event is removed from the queue. These queue-based event services are typically used to coordinate work among a number of possible readers and writers when processing is happening asynchronously, such as in a mapreduce-based processing system.
In contrast, pipe-based event services typically do not record the events as they occur. A pipe-based system works like plumbing: when a particular channel is open, then events will flow to their destinations as quickly as possible, with different systems providing guarantees such as guaranteed delivery or in-order delivery. Events do not stay in the channel until they are claimed by an endpoint; if there is no reader ready to receive the event, then the event is discarded. Pipe-based event services are typically used to coordinate real-time or high-volume work, such as in a real-time operating system.
Existing event services include various implementations using the Advanced Message Queuing Protocol (AMQP). AMQP is an open application layer protocol for message queuing that defines the interaction between the client and a server or broker. The protocol defines message exchanges for accessing and storing messages on various types of queues. The protocol also allows the client and server to define custom behaviors for the queue, such as message generation, message persistence, and message routing. Example implementations of the AMQP protocol include RabbitMQ and Apache Qpid.
Other existing event services may use the Extensible Message and Presence Protocol (XMPP). XMPP is an open protocol commonly used in instant messaging applications such as Google Talk and Facebook chat. The protocol involves one client sending an event to an associated server. The event contains addressing information for a destination client. The server examines this information and sends the event to the server associated with the destination client. If the destination client is online, the message is delivered. If the client is offline, the message is stored for later delivery. XMPP has also been used as a message delivery service.
Finally, PubSubHubbub (PuSH) is an open protocol for distributed event publishing an subscription. Based on Atom, PuSH aims to provide real-time change notifications without a client having to poll a server. PuSH uses long-polling in HTTP, which can use up available resources as the number of clients grows.
The following disclosure describes several embodiments of alternate solutions to the problem described above, some of which leverage, include, combine or modify the products and standard listed above.